High Performance

For a smooth video experience, we suggest increasing the framerate above the default of 30. This is especially important for high speed fitness or sports use cases.

Quickpose supports any framerate your camera supports and gracefully scales down to match available resources, although for best performance we recommend matching the camera framerate to meet the device's actual performance.

QuickPose provides a combination of performance options to choose from, the results typically being a balance between device compatibility, quality and speed:

Minimum Device Requirements

QuickPose supports older devices surprisingly well, achieving 40fps on the iPhone 8 on our lightest model:

We'd suggest the iPhone 8 as a sensibile minimum as it uses the Apple A11 Bionic chip. If you wish to support older devices, considering running QuickPose on selected images rather than real time video.

Changing the Camera Framerate

We recommend reviewing your device's camera framerate use to create a fluid and responsive experience for your users.

To customize the framerate using our helper classes, being by extending the recommended screen setup. Declare an additional state to hold the framerate, Nil sets it to the default camera fps which is typically 30.

@State private var frameRate: Double? = nil

This binding can be passed into the QuickPoseCameraView or QuickPoseCameraSwitchView if you want users to swap cameras easily:

ZStack(alignment: .top) {
    QuickPoseCameraView(useFrontCamera: true, delegate: quickPose, frameRate: $frameRate)
    QuickPoseOverlayView(overlayImage: $overlayImage)
}

We use a binding so that the frameRate can be changed at runtime, say if you were running on a legacy device or getting particularly bad output frames per second at run time you can update this accordingly. Bear in mind the camera framerate will be visible to the user, as the rendered cameraview will react to this value and be slower or more fluid depending on the framerate change.

Reviewing the output fps

It's important to review the output framerate as this is the actual performance your users will experience.

quickPose.start(features: [.showPoints()], onFrame: { status, image, features, feedback, landmarks in                
    switch status {
        overlayImage = image
        if case .success(let performance, _) = status {
            print(performance.fps, performance.latency)
        }
        ...
    }
})

The output framerate is how many frame per second were shown to the user, and the latency is the time taken for that frame to process, including the drawing of the overlays.

Here are the output fps for different camera frame rates on an iPhone 14 pro:

The fastest here is the 90fps camera. Note: even though the camera framerate of 120fps was requested, the actual output fps was only around 60fps!

tip

Bear in mind that requesting a high framerate can reduce performance if the device becomes overwhelmed with camera images. If the output framerate is lower than the camera framerate, you're likely to be doing more work than you need to and frames are being dropped. This is not a critical problem but would potentially slow other processing and drain battery life.

Performance Tip 1: Consider Post Processing

Our major performance tip for QuickPose is to consider post processing. By processing the frames after the event, all of your devices resources can be used to capture footage and process them, plus the added benefit of post processing being lag free.

Lag-free 240fps video rendering

Checkout our Post Processing Docs, but note this might not be applicable for all use cases, say where the user requires continuous feedback.

Performance Tip 2: Remove unused features:

Our big performance tip for QuickPose is to drop detailed hand and face features when you don't need them.

To do this, create a QuickPose.ModelConfig file and pass it to the quickPose.start method.

let modelConfigNoDetailedFaceOrHands = QuickPose.ModelConfig(detailedFaceTracking: false, detailedHandTracking: false)
quickPose.start(features: [.showPoints()], modelConfig: modelConfigNoDetailedFaceOrHands, onFrame: { status, image, features, feedback, landmarks in                
    switch status {
        overlayImage = image
        if case .success(let performance, _) = status {
            print(performance.fps, performance.latency)
        }
        ...
    }
})

You can also change this at runtime, using quickPose.update so you can turn these features on and off depending on your UX flow:

let modelConfigNoDetailedFaceOrHands = QuickPose.ModelConfig(detailedFaceTracking: false, detailedHandTracking: false)
quickPose.update(features: [.showPoints()], modelConfig: modelConfigNoDetailedFaceOrHands)

Performance Tip 3: Reduce model Complexity:

If your aiming to improve your current camera framerate, you can reduce your model complexity to have a big impact on output framerate:

Bear in mind these performance tweak only applied if the framerate is a challenge for the device, on the iPhone 14 Pro it can handle 40fps on all model complexities.

To do this, create a QuickPose.ModelConfig file and pass it to the quickPose.start method.

let modelConfigLight = QuickPose.ModelConfig(modelComplexity: .light)
quickPose.start(features: [.showPoints()], modelConfig: modelConfigLight, onFrame: { status, image, features, feedback, landmarks in                
    switch status {
        overlayImage = image
        if case .success(let performance, _) = status {
            print(performance.fps, performance.latency)
        }
        ...
    }
})

You can validate the model used by reviewing the output log:

QuickPose 1.2.0 started with MediaPipe version v0.8.10
I20230620 16:24:39.446619 1812983808 resource_util_apple.cc:78] Successfully loaded: pose_detection.tflite
I20230620 16:24:39.446753 1812410368 resource_util_apple.cc:78] Successfully loaded: hand_landmark_full.tflite
I20230620 16:24:39.446784 1814130688 resource_util_apple.cc:78] Successfully loaded: face_landmark.tflite
I20230620 16:24:39.446702 1814704128 resource_util_apple.cc:78] Successfully loaded: hand_landmark_full.tflite
I20230620 16:24:39.446764 1813557248 resource_util_apple.cc:78] Successfully loaded: pose_landmark_full.tflite
....

Changing model complexity requires restarting QuickPose

Unlike the detailed face and hand tracking, model complexity currently cannot be changed per frame.

Performance Tip 4: Keep the default resolution

One might expect lowering the resolution would improve performance but it doesn't have a significant impact and has a few downsides:

1. Customizing the resolution can drastically reduce device support and also affects the available frame rates.
2. Overlays are shown in the resolution of the camera feed, so the size of our overlays can change drastically if lower resolutions are used.
3. The same resolution is used by the displayed camera, and users are used to seeing themselves in HD.

Under testing we also note there isn't a clear improvement: